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Abstract —  Current  advances  operational  information  fusion  systems 
(IFSs)  require  common  semantic  ontologies  for  collection,  storage, 
and  access  to  multi  intelligence  information.  One  example  is  the 
connections  between  physics-based  ( e.g .  video)  and  text-based  (e.g. 
reports)  describing  the  same  situation.  Situation,  user,  and  mission 
awareness  are  enabled  through  a  common  ontology.  In  this  paper,  we 
utilize  the  uncertainty  representation  and  reasoning  evaluation 
framework  (URREF)  ontology  as  a  basis  for  describing  wide-area 
motion  imagery  (WAMI)  analysis  to  determine  uncertainty  attributes. 
As  part  of  the  Evaluation  of  Technologies  for  Uncertainty 
Representation  Working  Group  (ETURWG),  both  the  URREF  and  a 
WAMI  challenge  problem  are  available  for  research  purposes  from 
which  we  describe  the  URREF,  a  exemplar  schema  to  link  physics- 
based  and  text-based  uncertainty’  representations,  and  explore  an 
example  from  WAMI  exploitation  for  a  common  uncertainty 
demonstration. 
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I.  Introduction 

Semantic  ontologies  [1]  enable  a  framework  for  many 
applications  such  as  command  and  control,  emergency 
response,  and  infonnation  sharing  [2].  Infonnation  sharing,  and 
the  inherent  policies  within  an  architecture,  enable  data  to  be 
fused  into  actionable  knowledge.  A  key  to  infonnation  fusion  is 
to  reduce  uncertainty  that  may  come  from  many  sources  that 
require  a  unified,  common,  and  standardized  semantic 
understanding.  Figure  1  [3],  shows  the  relations  between 


sensed  and  reported  world  information  from  which  uncertainty 
representation  and  uncertainty  reasoning  are  required  for 
machine  processing  and  user  interaction,  refinement,  and 
understanding  [4,  5,  6,  7]. 

The  evaluation  of  how  uncertainty  is  processed  is 
dependent  on  the  system-level  metrics  such  as  timeliness, 
accuracy,  confidence,  throughput,  and  cost  [8],  which  also  are 
infonnation  fusion  quality  of  service  (QoS)  metrics  [9].  Future 
large  complex  information  fusion  systems  will  require 
performance  evaluation  [10]  and  understanding  of  the 
connections  between  various  metrics  [11],  It  is  a  goal  of  the 
Evaluation  of  Technologies  for  Uncertainty  Representation 
Working  Group  (ETURWG)  to  formulate,  test,  and  evaluate 
different  methods  of  a  semantic  uncertainty  ontology  that  is 
common,  universal,  and  standardized. 

Infonnation  fusion  system-level  metrics  include  timeliness 
(how  quickly  the  system  can  come  to  a  conclusion  within  a 
specified  precision  level),  accuracy  (where  can  an  object  be 
found  for  a  specified  localization  level),  and  confidence  (what 
level  of  a  probability  match  for  a  defined  recall  level). Clearly, 
different  choices  in  uncertainty  representation  approaches  will 
affect  the  achievable  timeliness,  accuracy,  and  confidence  of  a 
system,  and  therefore  must  be  considered  when  evaluating  both 
the  system’s  performance  as  a  whole  [12]  and  the  specific 
impact  of  the  uncertainty  handling  approach.  Yet,  when 
evaluating  timeliness  (or  any  other  system-level  metrics),  one 
will  likely  find  some  factors  not  directly  related  to  the  handling 
of  uncertainty  itself,  such  as  object  tracking  and  identification 
report  updates  (i.e.,  Level  1  fusion)  [13,  14,  15],  situation  and 
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Figure  1  -  Boundaries  of  the  Uncertainty  Representation  and  Reasoning  Evaluation  Framework  [3]. 
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threat  assessment  relative  to  scenario  constraints  (i.e.,  Level  2/3 
fusion)  [16],  overall  system  architecture  (e.g.  centralized, 
distributed,  etc.)  [12],  data  management  processes  and 
feedback  /  input  control  processes  (i.e.,  Level  4  fusion 
considerations)  [17],  and  user-machine  coordination  based  on 
operating  systems  (i.e.,  Level  5  fusion)  [18],  and  others. 

Key  to  the  various  Data  Fusion  Information  Group  (DFIG) 
[19]  levels  of  information  fusion  is  evaluation.  For  example, 
there  have  been  efforts  in  comprehensive  tracking  [20,  21], 
object  classification  [22],  and  situation  awareness  evaluation 
[23]  which  focus  on  measures  of  performance  (MOPs).  Future 
evaluations  will  include  high-level  information  Measures  of 
Effectiveness  (MOEs)  [24]  that  include  uncertainty 
characterization  [25], 

The  ETUWRG  has  developed  both  an  uncertainty 
framework  ontology,  but  also  has  a  series  of  use  cases.  One  use 
case  is  that  of  Wide-Area  Motion  Imagery  (WAMI)  for 
developments  in  Level  1  fusion  [26,  27,  28,  29],  Other 
computer  vision  working  groups  [30]  are  exploring  semantic 
technology  with  datasets  that  are  not  necessary  focused  on 
uncertainty,  but  have  a  rich  set  of  ontologies  and  datasets  for 
collaboration  and  comparisons. 

The  paper  investigates  the  URREF  for  WAMI  tracking. 
Section  II  explores  the  issues  of  uncertainty  characterization 
and  Section  III,  the  uncertainty  evaluation  framework.  Section 
IV  presents  a  WAMI  tracking  use  case  using  the  URREF  for 
timeliness,  accuracy,  and  confidence.  Section  V  provides  and 
discussion  and  Section  VI  conclusions. 

II.  The  Uncertainty  Representation  Problem 

The  In  format  ion  Fusion  community  envisions  effortless 
interaction  between  humans  and  computers,  seamless 
interoperability  and  information  exchange  among  applications, 
and  rapid  and  accurate  identification  and  invocation  of 
appropriate  services.  As  work  with  semantics  and  services 
grows  more  ambitious,  there  is  increasing  appreciation  of  the 
need  for  principled  approaches  to  representing  and  reasoning 
under  uncertainty.  Here,  the  term  "uncertainty"  is  intended  to 
encompass  a  variety  of  aspects  of  imperfect  knowledge, 
including  incompleteness,  inconclusiveness,  vagueness, 
ambiguity,  and  others.  The  term  "uncertainty  reasoning"  is 
meant  to  denote  the  full  range  of  methods  designed  for 
representing  and  reasoning  with  knowledge  when  Boolean 
truth-values  are  unknown,  unknowable,  or  inapplicable. 
Commonly  applied  approaches  to  uncertainty  reasoning 
include  probability  theory  [31],  expert  systems  [32],  fuzzy 
logic,  subjective  logic  [33,  34],  Dempster-Shafer  theory,  DSmT 
[35],  and  numerous  other  techniques. 

To  illustrate  the  challenges  of  evaluating  uncertainty 
representation  and  reasoning  in  information  systems,  we 
consider  below  a  few  reasoning  challenges  faced  within  the 
World  Wide  Web  domain  that  could  be  addressed  by  reasoning 
under  uncertainty  [1].  Uuncertainty  is  an  intrinsic  feature  of 
many  of  the  required  tasks,  and  a  full  realization  of  the  World 
Wide  Web  as  a  source  of  processable  data  and  information 
management  services  [3]  demands  formalisms  capable  of 
representing  and  reasoning  under  uncertainty  such  as: 


•  Automated  agents  (e.g.,  to  exchange  Web  information) 

•  Uncertainty’-laden  data,  (e.g.,  terrain  information) 

•  Non-sensoiy  collected  information  (e.g.,  human  sources). 

•  Dynamic  composability  (e.g.,  Web  Services). 

•  Information  extraction  (e.g.,  indexing  from  large  databases) 

These  problems  are  all  related  with  information  fusion, 
involve  both  textual-based  [36]  and  physics-based  [37]  data, 
and  can  be  easily  extrapolated  to  represent  the  more  general 
classes  of  problems  found  in  the  sensor,  data,  and  information 
fusion  domain.  A  recent  example  of  hard-soft  fusion  uses  a 
controlled  natural  language  (CNL)  for  data-to-decisions  [38]. 

III.  The  Uncertainty  Evaluation  Framework 

The  uncertainty  representation  and  reasoning  evaluation 
framework  (URREF)  includes  both  hard  sources  (e.g.  imaging, 
radar,  video,  etc.)  and  soft  sources  (HUMINT  reports,  software 
alerts,  etc.)  which  requires  integration  for  uncertainty  MOEs. 

Effectiveness  relates  to  a  system’s  capability  to  produce  an 
effect.  Many  benefits  of  fusion  include  providing  locations  of 
events,  extending  coverage,  and  reducing  ambiguity  and  false 
alarms.  The  goal  for  the  IFS  is  to  support  users  in  their  tasks 
whether  providing  refined  information,  reducing  time  and 
workload,  or  determining  completeness,  accuracy,  and  quality 
in  task  completion.  Effectiveness  includes  efficiency,  doing 
things  in  the  most  economical  way  (good  input  to  output  ratio), 
efficacy,  getting  things  done,  (i.e.,  meeting  objectives), 
correctness',  doing  "right"  things,  (i.e.,  setting  right  thresholds 
to  achieve  an  overall  goal  -  the  effect).  The  MOEs  support 
system-level  management  and  design  verification,  validation, 
testing,  and  evaluation.  The  URREF  output  step  involves  the 
assessment  of  how  information  on  uncertainty  is  presented  to 
the  users  and,  therefore,  how  it  impacts  the  quality  of  their 
decision-making  process. 

Key  aspects  of  measuring  effectiveness  come  from  quality 
of  service  (QoS)  metrics  that  can  be  utilized  for  hard-soft 
semantic  information  fusion  [39,  40,  41,  42]  Another 
perspective  includes  quality  of  information,  or  rather 
information  quality  (IQ),  metrics  to  combine  different  types  of 
uncertainty  to  an  established  quality.  IQ  metrics  establish  user 
semantic  content  as  a  schema  or  ontology  [43]  of  uncertainty 
analysis  such  as  a  popular  method  of  probabilistic  ontologies 
[44].  Together,  these  metrics  and  representations  support  a 
formal  theory  of  high-level  information  fusion  [2,  45]. 

The  URREF  ontology,  whose  main  concepts  are  depicted  in 
Figure  2  below  for  the  uncertainty  of  a  Thing,  is  a  first  step 
towards  building  a  semantic  standard.  The  core  of  the  ontology 
is  the  Criteria  class,  which  drives  the  development  of  the 
elements  of  the  subclasses  (Section  B).  The  Uncertainty 
Classes  were  either  taken  or  adapted  from  the  Uncertainty 
Ontology  developed  by  the  W3C’s  URW3-XG  [1].  The 
ontology  must  also  be  used  as  a  high-level  reference  for 
defining  the  actual  evaluation  criteria  items  that  will  comprise  a 
comprehensive  uncertainty  evaluation  framework.  Other  main 
class  definitions  include: 


•  A  source  class  is  the  origin  of  the  information.  A  physical 
sensor  is  one  important  example  of  a  source;  where  natural 
language  inputs  from  a  human  is  another. 


Figure  2  -  The  URREF  ontology  main  classes. 

•  A  Sentence  class  captures  an  expression  in  some  logical 
language  that  evaluates  to  a  truth-value  (e.g.,  formula, 
axiom,  assertion). 

•  A  Uncertainty’  Derivation  class  refers  to  the  way  it  can  be 
assessed  which  is  decomposed  into: 

1)  Objective  Subclass :  (e.g.,  factual  and  repeatable 
derivation  process). 

2)  Subjective  Subclass:  (e.g.,  a  subject  matter  expert's 
(SME’s)  estimation). 

•  A  Uncertainty  Model  class  contains  information  on  the 
mathematical  theories  for  the  representing  and  reasoning 
with  the  uncertainty  types. 

A.  Uncertainty  Type  Class 

Uncertainty  Type  is  a  concept  that  focuses  on  underlying 
characteristics  of  the  information  that  make  it  uncertain.  Its 
subclasses  are  Ambiguity,  Incompleteness,  Vagueness, 
Randomness,  and  Inconsistency,  all  depicted  in  Figure  3  below. 
These  subclasses  were  based  on  the  large  body  of  work  on 
evidential  reasoning  by  David  Schum  [31]. 


Figure  3  -  URREF  Ontology:  Uncertainty  Type  Class. 


B.  Criteria  Class 

The  Criteria  Class  is  the  main  class  of  the  URREF  ontology, 
and  it  is  meant  to  encompass  all  the  different  aspects  that  must 
be  considered  when  evaluating  information  uncertainty 
handling  in  multi-sensor  fusion  systems.  Figure  4  depicts  the 
Criteria  Class  and  its  subclasses: 

1)  Input  Criteria:  encompasses  the  criteria  that  directly  affect 
the  way  evidence  is  input  to  the  system.  It  focuses  on  the 
source  of  input  data  or  evidence,  which  can  be  tangible 
(sensing  or  physical),  testimonial  (human),  documentary,  or 
known  missing. 

•  Relevance  to  Problem  assesses  how  a  given  uncertainty 

representation  is  able  to  capture  why  a  given  input  is 
relevant  to  the  problem  and  what  was  the  source  of  the 
data  request. 

•  Weight  or  Force  of  Evidence  measures  how  a  given 

uncertainty  representation  is  able  to  capture  the  degree 
to  which  a  given  input  can  affect  the  processing  and 
output  of  the  fusion  system.  Ideally,  the  weight  should 
be  an  objective  assessment  and  the  representation 
approach  must  provide  a  means  to  measure  the  degree 
of  impact  of  an  evidence  item  with  a  numerical  scale 
such  as  value  of  information  [24]. 

•  Credibility,  also  known  as  believability,  comprises  the 

aspects  that  directly  affect  a  sensor  (soft  or  hard)  in  its 
ability  to  capture  evidence.  Its  subclasses  are  Veracity’, 
Objectivity’,  Observational  Sensitivity,  and  Self- 
Confidence. 

2)  Representation  Criteria:  encompasses  the  criteria  that 
directly  affect  the  way  information  is  captured  by  and 
transmitted  through  the  system.  These  criteria  can  also  be 
called  interfacing  or  transport  criteria,  as  they  relate  to  how 
the  representational  model  transfers,  passes,  and  routes 
information  within  the  system. 

•  Evidence  Handling:  is  a  subclass  of  representation  criteria 

that  apply  particularly  to  the  ability  of  a  given 
representation  of  uncertainty  to  capture  specific 
characteristics  of  incomplete  evidence  that  are  available 
to  or  produced  by  the  system.  The  main  focus  is  on 
measuring  the  quality  of  the  evidence  by  assessing  how 
well  this  evidence  is  able  to  support  the  development  of  a 
conclusion.  It  has  subclasses  Conclusiveness, 
Ambiguousness,  Completeness,  Reliability’,  and 
Dissonance. 

•  Knowledge  Handling:  includes  criteria  intended  to  measure 

the  ability  of  a  given  uncertainty  representation  technique 
to  convey  knowledge.  Its  subclasses  are  Compatibility 
and  Expressiveness  (which  is  further  divided  into  the 
subclasses  Assessment,  Adaptability,  and  Simplicity) 

3)  Reasoning  Criteria:  contains  criteria  that  directly  affect  the 
way  the  system  transforms  its  data  into  knowledge.  These 
can  also  be  called  process  or  inference  criteria,  as  they  deal 
with  how  the  uncertainty  model  performs  operations  with 
information.  It  has  the  following  subclasses: 


•  Correctness  measures  of  the  ability  of  the  inferential 

process  to  produce  correct  results.  In  cases  where  there  is 
no  ground  truth  to  establish  a  correct  answer  (including  a 
simulated  ground  truth),  the  representation  technique  can 
still  be  evaluated  in  terms  of  how  its  answers  align  with 
what  is  expected  from  a  gold  standard  (e.g.  SMEs, 
documentation,  etc.). 

•  Consistency  assesses  of  the  ability  of  the  inferential  process 

to  produce  the  same  results  when  given  the  same  data 
under  the  same  conditions. 

•  Scalability  evaluates  how  a  representational  technique 

performs  on  a  class  of  problems  as  the  amount  of  data  or 
the  problem  size  grows  very  large.  Scalability  could  be 
broken  down  into  additional  sub-criteria. 

•  Computational  Cost  computes  the  number  of  resources 

required  by  a  given  representational  technique  to  produce 
its  results. 


same  concepts  with  different  names.  For  example, 
accuracy  sometimes  is  used  as  a  synonym  of  precision; 
sometimes  the  terms  are  used  with  different  meanings. 
Indeed,  accuracy  and  precision  can  be  inversely 
related.  As  one  makes  the  granularity  coarser,  one  can 
expect  that  the  system  will  have  a  better  accuracy. 
Precision  can  also  be  used  to  determine  bounds  on  the 
certainty  of  the  reported  result. 

•  Interpretation  refers  to  the  degree  to  which  the 
uncertainty  representation  and  reasoning  can  be  used  to 
guide  assessment,  to  understand  the  conclusions  of  the 
system  and  use  them  as  a  basis  for  action,  and  to 
support  the  rules  for  combining  and  updating 
measures. 

The  above  concepts  are  being  explored  within  the 
ETURWG,  which  is  making  use  of  this  ontology  (shown  in 
Figure  4)  to  support  the  development  of  uncertainty  evaluation 
criteria  for  a  set  of  information  fusion  use  cases.  The  interested 


•  Performance  includes  metrics  to  assess  the  contribution  of 
the  representational  model  toward  meeting  the  functional 
requirements  of  an  information  fusion  system.  Other 
system  architecture  factors  also  affect  these  metrics.  This 
criterion  is  divided  into  subclasses  Timeliness  and 
Throughput. 

4)  Output  Criteria  relates  to  the  system’s  results  and  its  ability 
to  communicate  it  to  its  users  in  a  clear  fashion.  It  has  the 
following  subclasses: 

•  Quality  serves  to  assess  the  informational  quality  of  the 
system’s  output.  It  includes  Accuracy  and  Precision  as 
subclasses.  It  is  common  to  see  in  the  literature  the 


reader  should  refer  to  the  group’s  website  for  more  specific 
details  (http://eturwgamu.edu).  Note  that  the  URREF  ontology 
is  not  supposed  to  be  a  definitive  reference  for  evaluation 
criteria,  but  simply  an  established  baseline  that  is  coherent  and 
sufficient  for  its  purposes.  This  approach  privileges  the 
pragmatism  of  having  a  good  solution  against  having  an  “ideal” 
but  usually  unattainable  solution.  For  instance,  a  definitive 
reference  would  involve  having  universally  accepted 
definitions  and  usage  for  terms  such  as  "Precision."  This  is 
clearly  infeasible.  The  approach  also  takes  into  consideration 
that  more  important  than  naming  a  concept  is  to  ensure  that  it  is 
represented  clearly  and  distinctly  within  the  ontology  so  to 
ensure  the  consistency  for  such  applications  as  hard-soft  fusion. 


To  assure  utility  and  acceptability  of  the  URREF  ontology, 
most  of  its  concepts  have  been  drawn  from  seminal  work  in 
related  areas  such  as  uncertainty  representation,  evidential 
reasoning,  and  performance  evaluation.  The  ontology  has  built 
on  the  URW3  uncertainty  ontology  [1].  Also,  the  structure  and 
viewpoint  adopted  in  the  ontology  development  have  been 
tuned  to  addressing  the  uncertainty  evaluation  problem  and  its 
associated  perspective  (e.g.  how  information  is  handled  within 
a  fusion  system).  Next,  we  present  simultaneous  tracking  and 
identification  application  using  the  URREF. 


accuracy.  While  the  entire  URREF  cannot,  and  should  not,  be 
considered  for  the  schema  updates,  as  a  message  passing 
service  for  the  ontology,  the  first  issue  is  to  calculate  possible 
uncertainty  metrics  that  could  go  into  the  schema. 

B.  Metrics  to  Suppor  the  URREF  Semantic  Ontology 

For  the  metrics  available  in  the  Cursor  on  Target  Schema,  we 
seek  uncertainty  measures  of  confidence,  accuracy,  and 
timeliness,  as  related  to  uid,  time,  and  point;  respectively. 


IV.  EXAMPLE  -WAMI 

Characterizing  the  uncertainty  in  IF  processes  is  not  a  new 
research  topic.  An  example  is  the  Semantic  Web  as  part  of  the 
Web  Ontology  Language  (OWL)  (http://www.w3.org/TR/owl- 
guide/).  OWL  operational  semantics  support  message  formats 
(e.g.  XML  schema)  and  protocol  specifications  for  an 
ontology  knowledge  representation.  With  a  knowledge 
representation,  uncertainty  reasoning  can  be  determined  from 
the  message  format. 

A.  Schema 

A  schema  for  image  processing  is  shown  in  Figure  5  for  the 
Cursor  on  Target  (CoT)  program  [46].  As  detailed,  the  schema 
provides  target  type  and  identification  (ID)  allegiance,  time 
stamps,  and  coordinate  locations  (much  as  the  DFIG  level  1 
object  assessment  information  of  object  track  and  ID 
information).  While  the  schema  is  simple,  and  worked  well 
[47],  for  purposes  of  information  transmission,  processing, 
exploitation,  and  dissemination,  future  developments  could 
include  uncertainty  fields  from  the  URREF  ontology.  It  is 
important  to  research  which  semantic  content  is  most  relevant 
for  operational  information  fusion  management  and  systems 
design. 


<?xml  version=' 1 . 0 '  standalone=' yes 1 ?> 

<event  version="2 . 0" 
uid=" J-01334" 
type= " a-h-A-M- F-U-M" 
time="2005-04-05Tll : 43:38. 07Z" 
start="2005-04-05Tll:43:38.07Z" 
stale="2005-04-05Tll:45:38.07Z"  > 

<detail> 

</detail> 

<point  lat="30 . 0090027"  lon="-85 . 9578735"  ce="45.3" 
hae="-42 . 6"  le="99.5"  /> 

</event> 

Figure  5  -  Cursor  on  Target  Schema  [46] 

In  order  to  determine  what  ontology  content  can  be  added  to 
such  a  message  passing  schema,  there  are  three  issues  (1)  what, 
(2)  how  much,  and  (3)  which  ones.  For  the  case  of  physics- 
based  (video)  and  textual-based  reports,  we  need  to  determine 
what  semantic  content  could  be  useful.  One  simple  case  is  that 
either  a  human  analyst  can  report  a  “friendly”  in  the  uid  field, 
or  a  machine  tracker  could  extract  the  information  from  the 
video  to  update  the  uid  field  of  “friendly”.  One  example  of 
“friendly”  could  be  from  extracted  text  and  video  exploitation 
of  a  blue  vehicle.  What  is  obviously  missing  from  the  CoT 
schema  is  some  notion  of  uncertainty  with  the  measurements 
and  information  as  to  the  confidence,  timeliness,  and  position 


•  Credibility  /  Confidence:  evaluates  the  ability  to  discern  an 
object  based  on  a  known  target.  Classification  is  the 
target  match,  while  identity  is  target  allegiance.  If  targets 
are  of  known  entities,  it  can  be  assumed  that  the  targets 
not  classified  could  pose  an  ID  uncertainty.  Using  a 
Bayesian  approach  for  this  example,  we  determine  the 
relative  probability  from  the  likelihood  values  of  the 


object,  versus  of  target  clutter  £0 1  c ,  where  c  j  is  for/'  =  1, 
...,  n  clutter  types: 


Pro  |  c  = 


F  l  QIC  1 

^  cj  eC  [  -V  0|cj  ] 


(1) 


Using  plausibility,  uncertainty  is  everything  unknown 


UL=1-Pr0\e  (2) 

•  Timeliness:  evaluates  when  the  system  knows  enough 
information  to  make  a  decision  versus  when  it  was 
collected.  For  the  purpose  of  this  analysis  we  simulate 
the  deadtime  for  an  input  time  delay  (TD)  for  a  decision 
i,  as  related  to  the  user  achieving  a  control  decision  [48], 
Likewise,  in  the  action  selection  requires  time  as 
modelled  as  an  output  time  delay  ( TO\ ).  The  updated 
state-space  representation  is: 

x(?)  =  —A  x(t)  +  B  u(t  -  TD) 

(3) 

y(0  =  C  x(t  -  TO)  +  D  u(t) 

To  determine  the  estimation  parameters  of  A  and  B,  as  well 
as  the  output  analysis  of  C  and  D,  we  model  the  importance 
of  the  information  processing  as  related  to  the  cognitive 
observe-orient-decide-act  (OODA)  functions.  Uncertainty 
is  defined  as  the  decision  time  difference  of  arrival: 


UT  =  x(t  -  TO)  -  x(f  -  TD 0  (4) 

•  Accuracy:  evaluates  how  the  real  world  track  estimates 
from  the  measurements  compare  to  the  ground  truth.  For 
the  puipose  of  this  analysis,  the  real  world  is  reduced  to 
a  specified  track  estimate  xM,  as  related  to  ground  truth 
Xj.  Using  a  root-mean  square  error,  we  have: 

UL  =  yJ(xM-xTf  +  (vm-Jt)2  (5) 


Track  Accuracy 


Accuracy  can  be  determined  versus  the  ability  to  track  a 
target  exactly  :  1  -  UL.  Other  aspects  could  include  track 
purity  for  track-to-track  association  [49]  for  situation 
awareness  including: 

•  Specificity:  evaluates  how  much  of  the  real  world  clutter  is 

reduced  such  as  reducing  the  false  alarms.  While  we  do 
not  simulate,  we  can  deduce  from  the  track  confidence. 

•  Situation  Completeness:  evaluates  how  much  of  the  real 

world  the  system  knows.  For  the  purpose  of  this  analysis 
the  real  world  is  reduced  to  a  specified  region  of  space 
(the  volume  of  interest,  VOI)  during  a  given  time 
interval  (the  time  interval  of  interest). 

C.  Wide  Area  Motion  Imagery  Example 

For  the  ETURWG,  a  use  case  is  available  for  the  purposes  of 
semantic  investigation.  WAMI  has  gained  in  popularity  as  it 
affords  advanced  capabilities  in  persistence,  increased  track 
life,  and  situation  awareness,  but  it  also  poses  new  challenges 
[50,  51]  such  as  low  frame  updates  (timeliness). 

Leveraging  developments  from  computer  vision  [52,  53,  54, 
55],  methods  are  being  applied  as  part  of  the  ETUWG  [27-29]. 
The  persistence  coverage  affords  such  methods  as  multiple 
object  and  group  tracking  [56,  57,  58],  road  assessment  and 
tracking  [59,  60],  contextual  tracking  [61,  62],  and  advances  in 
particle  filtering  [59,  63].  Because  of  the  numerous  objects 
and  their  movements,  there  are  opportunities  for  linear  road 
tracking,  but  also  there  is  a  need  for  nonlinear  track  evaluation 

[64]  such  as  the  randomized  unscented  transform  (RUT)  filter 

[65]  for  accuracy  assessment.  These  issues  will  be  important 
for  future  work. 

We  utilize  the  results  from  a  WAMI  tracker  for  track 
location  accuracy,  the  pixels  on  target  for  classification  for 
target  identity  (e.g.  credibility),  and  the  timeliness  to  make  a 
decision.  We  are  tracking  four  targets  with  an  on-road  analysis 
with  a  nominated  target  of  interest,  as  shown  in  Figure  6. 
Vehicles  turning  off  road  are  not  considered  as  part  of  the  user 
defined  targets  of  interest.  Note:  the  entire  Columbus  Large 
Image  Format  (CLIF)  WAMI  image  collection  has  been 
presented  in  previous  papers  with  discussions  with  the  entire 
video  data  set  (see  the  ETURWG  website). 


Figure  6  -  WAMI  Tracking. 
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Figure  7  -  Target  Accuracy 

Figure  7  plots  the  target  accuracy  (which  is  the  inverse  of 
the  typical  plots  that  show  the  target  tracking  error).  Figure  8 
combines  the  track  accuracy  in  a  unified  display  plot  showing 
the  target  confidence  (uid)  and  the  accuracy.  The  confidence  is 
shown  as  solid  lines  and  the  timeliness  presented  as  the  black 
humps  where  the  time  intervals  are  shown  as:  orient  ( t  =  2.5- 
5),  observe  ( t  =  5-10),  decide  ( t  =  10-13)  and  act  ( t  =13-18) 
time  steps. 


Time  Response 


Figure  8  -  Confidence-Accuracy-Timeliness  Results. 

Using  the  above  information,  we  combine  the  credibility 
/confidence,  accuracy,  and  timeliness  (CAT)  for  a  semantic 
notion  of  fused  uncertainty  in  Figure  9  (where  the  normalized 
values  are  UT  =  Uc  +  Ur  +  UA)-  Together,  the  combined 
uncertainty  could  be  a  ontology  field  in  an  updated  CoT 
schema  to  give  the  user  a  quality  assessment  of  a  machine 
processed  semantic  representation  of  uncertainty. 

V.  Discussion 

Figure  9  shows  a  case  for  a  unified  uncertainty  estimation  and 
is  meant  for  discussion.  Given  the  choice  to  utilize  the  URREF 
ontology,  there  are  issues  associated  with  choosing  an 
ontology  representation  that  can  work  within  a  message 
passing  schema.  If  only  one  field  was  available,  say  ut,  then  is 
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it  appropriate  to  normalize  the  uncertainty  and  combine  for 
purposes  of  the  schema?  For  this  case,  only  one  target  was 
nominated  (like  the  CoT  program),  from  which  we  see  that  the 
combined  evidence  supports  a  reduction  in  uncertainty; 
namely  decreased  track  error,  increased  plausibility  and  hence 
ruling  out  the  uid  error,  and  the  timeliness  in  decision  making. 


CAT  Uncetainty  Plot 


Figure  9  -  Objective  Semantic  of  Uncertainty. 


VI.  Conclusions 

Characterizing  the  uncertainty  is  important  in  information 
fusion  (IF)  processes.  Evaluation  of  IF  systems  presents 
various  challenges  due  to  the  complexity  of  fusion  systems  and 
the  sheer  number  of  variables  influencing  their  performance. 
Developing  the  operational  semantics  will  include  issues  of 
representation,  reasoning,  and  policy  which  need  to  be 
considered  for  command  and  control  [66],  Representing 
uncertainty  has  an  overall  impact  on  system  performance  that  is 
hard  to  quantity  or  even  to  assess  from  a  qualitative  viewpoint. 
The  ETURWG  technical  considerations  unearthed  many  issues 
that  demand  a  common  understanding  that  is  only  achievable 
by  a  formal  specification  of  the  semantics  involved  [67,  68], 

In  the  paper,  we  utilized  the  current  URREF  ontology  in 
relation  to  an  established  schema  (Cursor  on  Target)  to  support 
the  development  of  a  specific  use  case  for  wide-area  motion 
imagery  (WAMI)  simultaneous  tracking  and  identification.  We 
also  presented  a  visual  analytic  method  for  uncertainty  metrics 
and  analytics.  Future  work  includes  group  tracking,  activity 
analysis,  hard-soft  fusion,  and  contextual  understanding. 

More  specific  requirements  to  evaluate  a  set  of  use  cases 
and  associated  data  sets  designed  by  the  ETURWG  are 
accessible  through  our  webpage  [http://eturwg.c4i.gmu.edu]. 
Although  it  is  clear  that  the  URREF  ontology  is  not  a  definitive 
reference  for  all  types  of  information  fusion  activities,  it  has 
proven  to  be  a  discussion  towards  a  common  framework. 
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